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Assessing Digital Information Literacy in Higher Education: 
A Review of Existing Frameworks and Assessments With 
Recommendations for Next-Generation Assessment 

Jesse R. Sparks , 1 Irvin R. Katz , 1 & Penny M. Beile 2 

1 Educational Testing Service, Princeton, NJ 

2 University of Central Florida, Orlando, FL 


Digital information literacy (DIL) — generally defined as the ability to obtain, understand, evaluate, and use information in a variety 
of digital technology contexts — is a critically important skill deemed necessary for success in higher education as well as in the global 
networked economy. To determine whether college graduates possess the requisite knowledge and skills in DIL, higher education 
institutions must be able to administer and use results from valid assessments of DIL. In this paper, we provide a comprehensive review 
of existing definitions of this construct in major frameworks from higher education and the workforce and propose an operational 
definition of DIL. Next, we provide a review of existing assessments of information literacy and related constructs, including features of 
the assessments, construct alignment, and psychometric properties (i.e., reliability and validity evidence). Finally, we discuss challenges 
and considerations surrounding the design, implementation, and use of next-generation assessments of DIL. We offer this review as a 
resource for higher education institutions in selecting among existing assessments or in designing their own measures. 

Keywords Digital information literacy; information literacy; information problem solving; higher education; assessment; student 
learning outcomes 
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Today’s information environment is constantly evolving as new technologies are developed and adopted by greater num¬ 
bers of individuals. Adding to the complexity of this environment is the exponential growth in the volume of digital 
information being produced (Gantz & Reinsel, 2012; School of Information Management and Systems, University of 
California - Berkeley, 2003; SINTEF, 2013). IDC, a company that has been tracking and analyzing the growth of digital 
data since 2005, has estimated that from 2012 until 2020 the size of the digital universe will effectively double every 2 years 
(Gantz & Reinsel, 2012). SINTEF (2013), an independent research organization, added that 90% of all the world’s data, 
which includes commercial and crowd-sourced images, videos, e-mail, voice mail, and texting, has been generated over 
the past 2 years. Because widely available mobile devices allow consistent and continual access to information networks 
(Pew Research Center, 2015) with a few taps of a smartphone, tablet, or other device, users can share text, images, and 
video data with relative ease from across the globe. The unprecedented growth in digital information and proliferation of 
related information and communication technologies (ICT) has impacted every aspect of our personal and professional 
lives (Organisation for Economic Cooperation and Development [OECD], 2013b; United Nations Educational, Scientific, 
Cultural Organization [UNESCO], n.d.). The way we live and work has changed dramatically due to the escalating scope 
and complexity of the digital information landscape, and so has the set of skills needed to fully participate in and benefit 
from an economy based on digital information creation and communication. 

Employees must be able to effectively locate, evaluate, and use information to carry out workplace responsibil¬ 
ities (Bruce, 1999; Cheuk, 2002). Uses of information and related technologies can range from collaborating with 
colleagues to intellectually manipulating information for problem-solving and decision-making and communicating 
those findings to customers (Bruce, 1999). Cheuk (2002) described in a UNESCO white paper how a lack of these 
skills can have a deleterious effect on businesses. According to the report, employees need to create, access, organize, 
use, evaluate, package, and present information for various purposes at work, yet inefficiencies in these processes, 
partly due to employees’ lack of pertinent skills, are often costly. For example, in one study, employees reported that 
they do not know what resources are available to them and therefore waste valuable time looking at information from 
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inappropriate sources, which may also compromise the quality of their work (Cheuk, 2002). Similarly, De Saulles (2007) 
attributed loss of business income in the United Kingdom to inefficient searching for information. Others have reported 
that many workforce employees lack the skills to use information effectively (Head, 2012; Raish & Rimland, 2016; 
Thompson, 2003). 

Not only does the inability to use information effectively cost businesses billions of dollars, but it also may be associated 
with a lower quality of life. UNESCO (n.d.) claimed that competency in media and information skills is necessary for life 
and work today. The OECD (2013a) added that proficiency in problem-solving in technology-rich environments (defined 
as the ability to use digital technology “to acquire and evaluate information, communicate with others, and perform 
practical tasks” [Programme for the International Assessment of Adult Competencies [PLAAC] Expert Group in Problem 
Solving in Technology-Rich Environments [PS-TRE], 2009]), traditional literacy, and numeracy are key skills needed for 
21st century economies. Results of PIAAC, which was administered to approximately 166,000 adults from 24 countries, 
suggest that people with low proficiency in solving problems in digital contexts face a much greater risk of economic 
disadvantage and a higher likelihood of unemployment and poor health (OECD, 2013a). 

The lack of information skills proficiency is an overarching societal concern, in terms of impact on both personal and 
professional life. In particular, employers are concerned about the proficiency levels of new entrants into the workplace. 
When 318 employers were surveyed about the preparation of recent college graduates for the workforce, 72% of respon¬ 
dents indicated that educational institutions should place greater emphasis on the ability to locate, organize, and evaluate 
information from multiple sources as a higher education learning outcome (Hart Research Associates, 2013). Similarly, 
qualitative interviews with 23 US employers (representing almost 500,000 employees in total) conducted in 2012 revealed 
that the employers generally perceived recent graduates as being technology savvy but were dismayed by the employees’ 
limited ability to locate information to solve workplace problems (Head, 2012). There thus appears to be a disjunction 
between perceptions of fluency with digital technology and effectiveness of actual use. 

Reports from employers, combined with focus group studies in which recent college graduates reported that they feel 
unprepared to solve information problems in the workplace (Head, 2012), have influenced the higher education com¬ 
munity, including professional associations and regional accrediting bodies alike, to identify effective information skills 
as a core learning outcome. For example, the American Association for Higher Education and the Council of Indepen¬ 
dent Colleges endorsed the Information Literacy Competency Standards for Higher Education (IL standards; Association 
of College and Research Libraries [ ACRL], 2000), and the Council for the Accreditation of Educator Preparations (C AEP; 
2013) Accreditation Standards has acknowledged technology and digital learning as a learning theme that pervades all 
content areas. Similarly, a number of regional accrediting bodies have identified expectations for effective information 
use (Saunders, 2007), including the Western Association of Schools and Colleges (WASC) Senior College and University 
Commission (2013), which has incorporated information literacy as one of five core competencies. The higher educa¬ 
tion community has acknowledged that the ability to use information effectively is fundamental to success in a rapidly 
changing technology- and information-intensive society, in turn creating the need to establish baseline skill levels and 
assess student learning related to these outcomes. This need to assess digital information literacy (DIL) skills requires 
valid measures that meet the needs of higher education institutions. 

In the current paper, we investigate the construct of DIL, both from the perspective of possible definitions as well as 
through a review of available assessments of information literacy skills. Our purpose is to assist institutions in think¬ 
ing through the various assessment and measurement issues associated with evaluating the DIL skills of their students, 
such as for the purposes of accreditation or curricular improvement. First, we describe various definitions of DIL and 
related constructs put forward in the literature by higher education, workforce, and other organizations, including areas 
of commonality and discrepancy among these definitions. From this synthesis, we present a proposed operational defi¬ 
nition of DIL that can be used to support the design of assessments of DIL competency. We then review several existing 
assessments of DIL with respect to the assessment purpose, coverage of the construct, and item formats. We also include 
discussions of the reliability and validity evidence associated with each assessment, where available. Following this review, 
we discuss the challenges associated with designing DIL assessment. Finally, we present recommendations for assessment 
design features that can be used to support the development of assessments of DIL as a student learning outcome (SLO) 
for higher education. The review of current assessments and the proposed definition and assessment design presented 
here are intended to support higher education institutions in either developing their own local assessments or choosing 
among available alternatives when deciding to evaluate the DIL skills of their students. 
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Review of Frameworks of Digital Information Literacy 
Definitions of Digital Information Literacy in Key Frameworks 

DIL involves obtaining, understanding, evaluating, and using information in a variety of digital technology contexts. 
Extending beyond traditional definitions of “mere literacy” (cf. New London Group, 1996, p. 64) as a set of linguistic skills 
enabling reading (i.e., decoding and comprehension) and writing (i.e., transcription, composition) to build knowledge 
and communicate with others, DIL encompasses the knowledge and skills required for critically and effectively using 
digital information to achieve personal, civic, or workplace goals. However, definitions of this construct vary depending 
on the extent to which they emphasize particular dimensions of proficiency within DIL, such as a focus on use of specific 
technologies (e.g., use of e-mail or web browsers) versus the ability to critically evaluate and apply specific information 
content to answer questions or solve problems. For the purposes of designing an assessment of DIL as an SLO, examining 
the points of consistency and discrepancy among the definitions or frameworks for DIL helps ensure that the assessment 
will provide evidence of valued aspects of the construct (i.e., construct validity). Table 1 presents definitions of DIL drawn 
from frameworks developed at state, national, and international levels. These frameworks serve as a basis for our proposed 
operational definition of DIL to inform a DIL assessment that serves the needs of higher education institutions. 

Dimensions of Digital Information Literacy 

A review of the definitions presented in Table 1 indicates a great deal of similarity among the conceptions of DIL available 
in the existing literature, although some definitions emphasize certain aspects of the construct. Table 2 displays the cor¬ 
respondence between the frameworks reviewed and key dimensions of the DIL construct apparent in those frameworks. 
Across the definitions in the frameworks, the components of DIL include the use of digital technologies in the service of 
defining information needs; accessing, evaluating, and managing or organizing sources; integrating or synthesizing infor¬ 
mation from multiple sources; creating and communicating information to audiences; using or applying information to 
solve problems; and understanding of ethical and legal issues surrounding information use. These components appear as 
columns in Table 2. 

Of these dimensions, accessing, evaluating, and managing information were explicitly included in all or nearly all 
frameworks reviewed (and some frameworks refer to organization or management but consider it a subcomponent of 
another skill; e.g., International Society for Technology in Education [ISTE], 2007). Communication, application to 
problem-solving, and ethical considerations were also quite common, although some frameworks only indirectly refer 
to communication or information use (e.g., the PIAAC PS-TRE framework emphasizes problem-solving but does not 
include ethical or legal use of information; PIAAC Expert Group in PS-TRE, 2009). Defining and refining information 
needs, integrating information from multiple sources, and creating digital media products were mentioned as part of the 
construct, though with somewhat less consistency than other aspects. For example, only five frameworks explicitly refer 
to integration and synthesis skills, though four allude to these components in the context of applying information to solve 
problems. Interestingly, despite the fundamental integration of information and communication technology embedded 
in the notion of DIL, frameworks varied in the extent to which they explicitly included the use of digital technology tools 
and applications in their definition of the construct. 

In sum, DIL involves a variety of goal-driven interactions with information sources and products in digital contexts, 
including the ability to define and establish goals for information seeking and retrieval; successfully accessing relevant 
material; evaluating retrieved sources for their quality and reliability; organizing the information contained in those 
sources according to a scheme that suits one’s purposes; making sense of varied and potentially conflicting information 
by integrating across multiple sources; and using that integrated understanding to answer questions, solve problems, or 
create digital media products that make effective use of information. From this summary, we propose an operational def¬ 
inition of DIL, specifically intended for use in supporting the design of assessments of DIL as an SLO in higher education 
contexts. 

Proposed Digital Information Literacy Operational Definition 

Based on the previous synthesis of existing definitions of DIL, we propose to define DIL as the ability to function in 
a knowledge society through the appropriate use of information and communication technology to solve information 
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problems, including problem definition and data analysis; digital citizenship , understanding responsible and ethical use of technology and 
information; technology operations and concepts, basic understanding of how to select and make use of technology tools, including troubleshooting 
(ISTE, 2007). 
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Table 2 Alignment of Existing Frameworks With Dimensions of Digital Information Literacy 


Framework 

Define 

needs 

Access 

Evaluate 

Manage/ 

organize 

Integrate/ 

synthesize 

Create 

Communicate 

Apply to 
solve 
problems 

Ethical/ 

legal 

issues 

Use of 
digital 
technologies 

ACRL 2000 

X 

X 

X 





X 

X 


ACRL 2015 

X 

X 

X 

~ 


X 

X 

X 

X 


LEAP 

X 

X 

X 

X 

X 


X 

X 

X 


CETF 


X 

X 

X 

X 

X 

X 



X 

ATC21S 


X 

X 

X 

X 

X 


X 

X 

X 

ETS ICT LITERACY 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

ANZIIL 

X 

X 

X 

X 


X 

X 

X 

X 

~ 

PIAAC PS-TRE 

X 

X 

X 

X 

X 


X 

X 


X 

UNESCO MIL 

X 

X 

X 

X 


X 

X 

X 

X 

~ 

IC3GLOBAL 


X 


X 


~ 

X 


X 

X 

CITL FITNESS 1999 

X 

X 

X 

X 


X 

X 

X 


X 

ISTE STANDARDS-S 

X 

X 

X 

~ 

~ 

X 

X 

X 

X 

X 


Notes. X = explicitly covers this dimension of the construct; ~ = partially or indirectly covers this dimension. Blank cells indicate that 
the framework does not cover this dimension. 


Table 3 Components of Proposed Digital Information Literacy Construct Definition 


Component Description 


Define 

Access 

Evaluate 

Manage 

Integrate 

Create 


Understand and articulate the scope of an information problem to facilitate the search for information through 
digital technology 

Collect and/or retrieve information through appropriate use of digital technology 
Judge the extent to which digital information meets the needs of an information problem 
Organize digital information to help you or others find it later 

Interpret and represent information using digital technology to synthesize, summarize, compare and contrast 
information from multiple sources 

Adapt, apply, design or construct information through digital technology 


problems, including the ability to research, organize, and synthesize information through digital technology and having a 
fundamental understanding of the ethical/legal issues surrounding the use of such information. The proposed definition 
includes six components that are commonly included within existing frameworks: defining, accessing, evaluating, man¬ 
aging, integrating, and creating information. Each of these components is further defined in Table 3; these definitions 
parallel those found in the reviewed frameworks, especially California Emerging Technology Fund (CETF; 2008), the 
International ICT Literacy Panel (2002), and Educational Testing Service (ETS; Katz et al., 2004). 

Four of the dimensions listed in Table 2 are not explicitly part of our proposed definition: communicate, apply 
to solve problems, ethical/legal issues, and use of digital technologies. Although we recognize the importance of 
communication — such as expressing the results of one’s attempts to research and integrate information—through 
digital environments, assessments of communication skills are widely available, including through SLO assessments 
of writing proficiency (for a review of such offerings, see Sparks, Song, Brantley, & Liu, 2014). However, activities 
related to communication, such as assembling data displays or synthesizing multiple results into a conclusion, are 
included in the proposed definition under the categories of creating and integrating information, respectively. Sim¬ 
ilarly, application to problem-solving is encompassed under descriptions of integrating and creating information to 
achieve a goal. 

Ethical and legal use of information, while not a major dimension of the proposed DIL construct, is involved in the 
processes of integrating and citing information from sources, selecting publicly available material (versus copyrighted 
material) for an informational display, or other tasks that involve creation or integration of information products (e.g., 
data displays, infographics, executive summaries). Therefore, these skills are indirectly assessed by tasks requiring stu¬ 
dents to demonstrate source selection or integration. Further, it may be easier to assess specific ethical or legal concepts 
through multiple-choice items rather than through performance tasks where students must demonstrate ethical behav¬ 
ior, an issue that we will return to in our review of existing DIL assessments and associated design considerations. Finally, 
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although digital tools and environments form the context for assessing DIL skills, our proposed definition does not include 
knowledge of, or skills in using, particular digital technology, an issue we discuss in the last section of this report. 

Comparison of Proposed Definition With Existing Frameworks 

Our proposed definition includes several of the most prominent aspects of appropriate retrieval and use of information in 
the context of digital ICT. Defining information needs is common to all information literacy focused frameworks (ACRL, 
2000,2015; Bundy, 2004; OECD, 2013b; Rhodes, 2010; UNESCO, 2013) and the ISTE Standards for Students (ISTE, 2007). 
Accessing, evaluating, and managing information are important components of both information literacy-focused and 
more technology-focused frameworks. Integration is most clearly mentioned by the more synthetic frameworks (i.e., 
frameworks that themselves were based on syntheses of multiple construct definitions; Binkley et al., 2012; CETF, 2008; 
PLAAC Expert Group in PS-TRE, 2009; Rhodes, 2010). Finally, creating media products by using digital technologies 
appears in both technology-focused and synthetic frameworks. Further, the addition of objectives related to creating infor¬ 
mation in the recent ACRL Framework for Information Literacy for Higher Education (IL Framework; ACRL, 2015) — and 
the acknowledgement of “information creation as a process” (p. 6) — marks an important distinction from the previous IL 
standards (ACRL, 2000), which did not explicitly emphasize information creation or the use of digital technologies per se 
in the creation process. In contrast to frameworks that focus on understanding and use of digital technologies themselves 
rather than the cognitive activities required for their use (e.g., word processing, databases, or spreadsheets; Certiport, n.d.; 
ISTE, 2007), the proposed DIL framework emphasizes the problem-solving and reasoning skills that occur in the context 
of using technology to solve information problems (cf. Katz, 2005). Thus, our proposed definition of DIL is consistent 
with more recent, integrative conceptions of what it means to proficiently engage with the digital information landscape 
in the service of solving authentic information-based problems (ACRL, 2015; OECD, 2013a; UNESCO, 2013), including 
task definition, retrieval, source evaluation, information management, source integration, and creation of products that 
represent and express the integrated knowledge and proposed solutions one has assembled. 

The frameworks reviewed previously not only suggest learning outcome targets for higher education institutions, but 
several of them also have been used directly as the basis for the design of assessments of information literacy skill. 
Notably, the IL standards (ACRL, 2000) were used as the basis for the development of several assessments, including 
the Information Literacy Test (ILT; Madison Assessment, 2015), the Standardized Assessment of Information Literacy 
Skills (SAILS; Project SAILS, 2015a), and the Research Readiness Self-Assessment (RRSA; Ivanitskaya et al., 2007). The 
Australian Information Skills Survey (ISS; Catts, 2005) was developed to assess the six standards presented in the Aus¬ 
tralia and New Zealand Information Literacy (ANZIIL) framework. The OECD developed and administered the PS- 
TRE assessment based on the corresponding PLAAC framework (PLAAC Expert Group in PS-TRE, 2009); similarly, the 
technology-focused assessment Internet and Computing Core Certification (IC3; Certiport, n.d.) was based on the IC3 
Global Standard, developed by an expert panel (Global Digital Literacy Council [GDLC]) and informed by the ISTE (2007) 
Standards for Students. The ETS definition served as the foundation for the development of the scenario-based iSkills™ 
assessment (Katz et al., 2004). The Information Literacy Assessment & Advocacy Project (ILAAP, 2015) assessment is 
aligned to both the IL standards (ACRL, 2000) and the newly released IL framework (ACRL, 2015). That framework is 
also being used to inform the design of the Threshold Achievement Test of Information Literacy (TATIL, n.d.); however, 
at the time of this writing, the assessment is still in development, so we exclude it from the current review. 


Review of Existing Assessments of Digital Information Literacy 

We now turn to a review of existing assessments of DIL (or related constructs) that might be used to evaluate the 
DIL proficiency of higher education students. This review is intended to illuminate the advantages and disadvan¬ 
tages of current approaches to assessing DIL skills, in order to inform recommendations for selecting or designing 
an assessment intended to provide actionable information to support program evaluation and curricular improve¬ 
ment, or to provide individual feedback or certification of skills. First, we describe our rationale for the choice of 
assessments included in this review. Then, we review key features of each assessment in terms of its format, purpose, 
construct coverage, and evidence of reliability and validity, where available. Detailed information about each assessment 
appears in Table 4. 
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Internet and Certiport Professional 45 min per Computer Multiple-choice and Three exams: 45 items Automatically scored Construct informed by earlier 

Computing Core certification exam (web-based) survey items, plus per form for version of ISTE Standards, but 

Certification (IC3) performance-based computing focused on IT certification. 

items fundamentals, key Living onbne exam includes 
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iSkills Educational Program 60 min Computer Simulation tasks with Seven tasks composed Automatically scored; 

Testing evaluation (online) selected-response and of 3-5 min individual and 

Service performance-based interactions with group reports 

(ETS) items (use simulated software provided. 

appfications) 





J. R. Sparks et al. 


Assessing Digital Information Literacy in Higher Education 


Table 5 Correspondence of Dimensions of Proposed Digital Information Literacy (DIL) Construct Definition With Existing Assess¬ 
ments of DIL and Related Constructs 




Proposed DIL construct dimensions 



Other dimensions 


Assessment 

Define 

needs 

Access 

Evaluate 

Manage/ 

organize 

Integrate/ 

synthesize 

Create 

Communicate 

Apply 
to solve 
problems 

Ethical/ 

legal 

issues 

Use of 
digital 
technologies 

Type 1 

ILT 

X 

X 

X 


X 




X 

X 

SAILS 

X 

X 

X 






X 

X 

RRSA 

X 

X 

X 





X 

X 

X 

ISS 

X 

X 

X 

X 

X 

X 

X 

X 

X 


ILAAP 

X 

X 

X 

X 

X 




X 

X 

Type 2 

IC3 


X 


X 



X 


X 

X 

ECDL 

~ 

X 

X 

X 


X 

X 


X 

X 

Type 3 

CLA+ 



X 


X 


X 

X 



PIAAC PS-TRE 

X 

X 

X 

X 

X 


~ 

X 


X 

ISKILLS 

X 

X 

X 

X 

X 

X 

X 

~ 


X 


Notes. X= assessment measures this dimension; ~ = assessment indirectly or partially claims to measure this dimension. Blank cells 
indicate that the assessment does not provide evidence of this dimension. 


Selection of Current Assessments 

For the purposes of this review, we selected assessments with a focus on higher education or adult populations, span¬ 
ning various purposes (i.e., program evaluation and curricular improvement, individual certification, international 
benchmarking). Thus, we excluded from our review assessments of DIL competencies targeted at populations below 
postsecondary education, such as the Programme for International Student Assessment (PISA) Digital Reading Assess¬ 
ment (OECD, 2014) or the Online Research and Comprehension Assessment (ORCA; Leu et al., 2015). However, we 
do include the PIAAC PS-TRE component of the Survey of Adult Skills (OECD, 2013a, 2013b) because the assessment 
is designed to measure the information problem-solving skills of adults across a variety of authentic technological and 
task contexts (PIAAC Expert Group in PS-TRE, 2009). We also include professional certification tests such as the IC3 
and the European Computer Driving Licence (ECDL) 1 because these assessments are used in workforce contexts to 
signal employers that employees have a particular level of proficiency with DIL or facility with various technological 
environments. 


Purpose of the Assessments 

The purpose of an assessment has direct implications for how the assessment is designed and administered and how the 
resulting scores should be interpreted and used. Although our goal is to support the development of assessments of DIL as 
a higher education outcome, we examined assessments designed for a variety of purposes, including program evaluation 
and curricular improvement (ISS, ILT, ILAAP, Collegiate Learning Assessment-1- [CLA+]), individual-level certification 
or feedback (IC3, ECDL), assessments that provide both institutional and individual information (RRSA, SAILS, iSkills), 
and international benchmarking assessments (PIAAC PS-TRE). Some of the assessments (i.e., ECDL and CLA+) we 
included because their target constructs overlap with definitions of DIL, although they may not directly be derived from 
the frameworks listed in Table 1. 

Broadly, these assessments fall into three categories on the basis of format and focal constructs. Table 5 presents an 
analysis of the correspondence between the skills targeted in each assessment (i.e., as described in construct definitions or 
rubric statements) and the dimensions of the proposed DIL construct definition presented previously (see Table 3). The 
assessments reviewed are organized by the three categories defined previously; we discuss construct coverage with respect 
to assessments falling within each of these categories. 
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Information Literacy Focus With Multiple-Choice and Constructed Response Items: Type 1 Assessments 

First, we review several assessments designed to align with major frameworks for information literacy, which rely primar¬ 
ily on traditional item types, such as multiple-choice, true/false, or rating scales (i.e., self-report measures). This review 
includes four assessments directly informed by the ACRL IL standards (ACRL, 2000), namely, Madison Assessment’s ILT 
(Wise, Cameron, Yang, & Davis, 2009), SAILS (Project SAILS, 2015a), RRSA (Ivanitskaya, 2011), and ILAAP (Goebel, 
Knoch, Thomson, Willson, & Sharun, 2013). Of these, the ILT has the most limited construct coverage; multiple-choice 
items on this assessment align with four of the IL standards (ACRL, 2000), excluding application of information to solve 
problems, which may be more difficult to validly assess using the multiple-choice format. It is also only intended to pro¬ 
vide information about groups of students in the aggregate; no individual-level information is reported, so it cannot be 
used to provide diagnostic information or direct student feedback. In contrast, SAILS, RRSA, and ILAAP are aligned 
with the full span of IL standards (i.e., none are specifically excluded from the assessment scope). However, only SAILS 
and RRSA can provide individual-level feedback. SAILS regrouped the IL standards (ACRL, 2000) into eight skill sets 
that define the measurement targets in the assessment, including developing a research strategy; selecting search tools, 
using features of search tools, and successfully conducting searches; retrieving, evaluating, and documenting sources; and 
understanding economic, legal, and social issues of information use (Radcliff, Salem, O’Connor, & Gedeon, 2007). The 
SAILS test has two versions: a cohort version designed for program-level assessment and an individual version that pro¬ 
vides scores directly to students. The RRSA, which measures objectives aligned with the IL Standards (ACRL, 2000) as 
well as students’ self-reported beliefs and experiences related to information literacy, goes one step further and provides 
students with automated individualized feedback, including subscores for both finding and evaluating information, upon 
completing the test. The ILAAP (2015) is a freely available tool designed as a flexible solution for institutions to assess stu¬ 
dents’ information literacy skills; institutions can select any combination of questions from a pool of both multiple-choice 
content questions (aligned to the IL standards and IL framework) and survey items (which primarily evaluate students’ 
perceptions of library instruction received). 

The final instrument in this category is the ISS (Catts, 2005), which is aligned to the ANZIIL framework (Bundy, 2004). 
The assessment items are aligned with all six of the ANZIIL framework standards, although because the items require stu¬ 
dents to provide a self-report rating (e.g., “I only list resources I have actually referred to in my assignment bibliography”; 
Catts, 2005, p. 13), this assessment provides a more subjective rather than objective picture of students’ information liter¬ 
acy skills. Thus, the ISS provides group-level information on the extent to which students report engaging in a variety of 
skills associated with the ANZIIL framework standards (but this may over- or underestimate what surveyed students actu¬ 
ally do in the context of solving information problems). This critique similarly applies to multiple-choice and true/false 
item types, which can provide valid information about students’ knowledge of particular information literacy skills and 
concepts, but cannot provide much evidence of the extent to which students effectively deploy those skills in the various 
contexts of their use. As is evident in Table 5, the assessments with more traditional format and emphasis on established 
information literacy standards are more likely to emphasize knowledge of concepts and skills (principles for beginning 
research, for accessing and evaluating different kinds of information sources, or ethical and legal considerations) and less 
likely to emphasize reasoning and problem-solving (i.e., integration, creation, or application). 

Technology Literacy Focus With Multiple-Choice and Constructed Response Items: Type 2 Assessments 

A second type of assessment includes tests that primarily emphasize interaction with specific digital ICT. Two example 
assessments illustrate the technology-focused approach: Certiport’s IC3 and the ECDL exam. Certiport’s IC3 is a cer¬ 
tification exam built on the IC3 Global Standard, a set of objectives informed by the GDLC, an international panel of 
experts. The ECDL is an exam given to secondary school students worldwide. Since its inception in 1996, the exam has 
been administered to 14 million people (ECDL Foundation, 2015a). The exam is used in a variety of ways, primarily to 
measure the technology literacy skills of youths over time. The ECDL Foundation is responsible for reviewing content, 
currency, and validity (ECDL Foundation, 2015b). 

In terms of construct coverage, both of these assessments measure skill in accessing, organizing, and communicating 
with digital technologies, including an emphasis on ethical and legal use. For example, the base modules of ECDL include 
computing fundamentals (basic use of digital technologies), online essentials (encompassing accessing, evaluating, man¬ 
aging, communicating, and responding ethically and appropriately to information in web browsing and e-mail contexts), 
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word processing, and spreadsheets (creating files and media products, using features of the technology). Students can 
opt to take the four base modules, plus a combination of standard or advanced modules on specific applications (e.g., 
databases, presentation software). In contrast, the IC3 exam consists of three modules: key applications, computing fun¬ 
damentals, and living online. The IC3 Living Online module overlaps the most with the D1L framework (see Table 5), 
including accessing, managing, and communicating via e-mail. Note that despite their use of simulated software envi¬ 
ronments, these assessments focus primarily on the use of the technology for its own sake, rather than the application of 
digital tools to achieve specific purposes or to solve problems. 

Both assessments include multiple-choice items, interactive items (moving objects on the screen), and performance- 
based tasks that require the use of simulated software. Both exams are “vendor neutral,” presenting simulated software that 
shares characteristics (menus and keyboard commands, functionality) of common software applications but are unique 
to the exam. These performance tasks allow for a variety of means of completing a task (e.g., printing a file); scoring is 
done automatically and is based on whether the candidate performed one of the allowable paths (i.e., a specific series of 
menu selections or keyboard presses) for the task. In fact, some items cover information literacy concepts not included 
in our framework, such as the ethical and legal use of information. However, the majority are technology-focused skills 
such as opening a file in a word processor, embedding a video in a presentation slide, navigating to a provided URL in a 
browser, or summing entries in a spreadsheet. 


Digital Information Literacy Focus With Performance-Based Tasks: Type 3 Assessments 

The third category of assessments reviewed includes those emphasizing both information literacy and technology-related 
skills, using more interactive, performance-based tasks as the primary format. This category includes the CLA+ (Klein, 
Benjamin, Shavelson, & Bolus, 2007), the PIAAC PS-TRE (OECD, 2013c), and the ETS iSkills assessment (Katz, 2007). 
First, we examine the CLA+, which is considered to be an assessment of higher education students’ critical thinking 
and written communication skills (Arum & Roksa, 2011; Klein et al., 2007; Liu, Frankel, & Roohr, 2014). However, we 
include this assessment in our review because (a) some of the objectives of the performance assessment task overlap with 
identified information literacy skills (e.g., evaluating sources) and (b) this assessment has been designed and validated for 
use as an SLO assessment (Klein et al., 2009). Specifically, the performance tasks present students with a document library 
including a range of digital sources that vary in type and content (e.g., memos, newspaper articles, research reports, data 
tables, graphs, or e-mails). Students are expected to evaluate the credibility and usefulness of the evidence provided in 
the sources and to analyze, synthesize, and draw conclusions from this evidence in their response to the task (Council 
for Aid to Education, 2014). Because these measurement targets overlap with our proposed components of evaluation, 
integration, and creation of information, and because validity evidence exists for the CL A+, we maintain it as an important 
point of comparison in evaluating the advantages and disadvantages of particular assessment design approaches (in this 
case, a performance-based task with a single constructed response format item). In terms of construct coverage, this 
assessment explicitly covers several skills related to proficient use of information sources, including evaluation, integration 
and synthesis, communicating, and applying the information to solve problems. While the sources are presented in a 
digital format, students are not asked to make use of technology or demonstrate proficiency with ICT tools in the context 
of the CLA+; rather, the emphasis is on students’ ability to compose a well-formed written response that includes accurate 
analysis of provided source materials. Thus, some inferences about students’ interactions with digital information sources 
could be made on the basis of CLA+ results, although this is not the primary construct being measured. 

In contrast to the completely open-ended approach, the iSkills and PS-TRE assessments present performance-based 
tasks that leverage simulated software environments, requiring test takers to interact with simulated components and 
select appropriate responses. For PS-TRE, items are designed according to three proficiency levels, where the highest level 
requires multiple steps, use of varied technologies, and deployment of higher-order cognitive processes such as monitor¬ 
ing and integration (OECD, 2013c). For example, one relatively simple sample item requires test takers to categorize a 
number of e-mail messages into folders; a more complex example might require test takers to manage a meeting room 
reservation system, including identifying available times that meet requester requirements, declining requests that can¬ 
not be filled, and integrating information across two simulated applications (e-mail and a web-based reservation system; 
OECD, 2013c). In terms of construct alignment, the PS-TRE assessment measures many critical aspects of DIL, including 
defining information needs, accessing, evaluating, managing, and integrating information, and most importantly for this 
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particular assessment, the application to problem-solving in the context of digital technology applications. The design of 
the PS-TRE assessment tasks is similar in some ways to the iSkills assessment developed by ETS. 

The iSkills assessment is a scenario-based performance assessment of students’ digital literacy skills. The assessment 
presents students with a series of 12 tasks, each of which consists of (a) the description of a situation in which information 
must be handled using digital technology and (b) simulated digital tools (e.g., word processor, presentation software) that 
students use in their solution to the task. The assessment system records student interactions with the simulated tools and 
automatically scores students’ responses as well as solution process metrics that, through field studies and other empirical 
research, have been shown to be construct relevant. The scenarios cover a range of workplace, academic, and personal 
uses of information in the context of technology. Students might be asked to manage e-mails from a boss, complete work 
on integrating search results for a group research project from a class, or help some family members to find information 
about their father’s high school garage band. Each realistic scenario motivates the location, use, or creation of information 
and introduces the simulated digital tools needed to complete the 3- to 5-minute task. 

Overall, iSkills provides evidence of many key dimensions of DIL skill—including the six dimensions featured in our 
proposed definition of DIL (define, access, evaluate, manage, integrate, and create; see Table 3), plus an added dimension 
of communicating about and with digital information. The performance-based assessments reviewed previously pro¬ 
vide richer tasks that attempt to simulate realistic contexts of information use; these assessments vary in the use of item 
types (i.e., open-ended constructed response vs. interactive tasks), but each achieves more direct assessment of examinee’s 
application of DIL knowledge and skills. The extent to which this richer assessment format yields enhanced reliability and 
validity as compared to other assessment formats is discussed in the following section. 

Reliability and Validity Evidence 

A review of published reliability and validity evidence for the information literacy assessments discussed previously 
appears in Table 6. In the following sections, we summarize the available validity evidence with respect to reliability and 
various types of validity: content, construct, concurrent, and predictive validity. Note that across the assessments reviewed, 
the amount of available validity evidence varies considerably, and there is variation in the level of details that are reported 
(e.g., some reports do not include important considerations such as sample size or actual values of correlation coefficients 
reported to be significant). When such figures were not indicated by the study authors, we note this. 

Reliability Evidence 

Reliability evidence reflects the extent to which a test consistently measures the same construct. One form of evidence 
is internal consistency reliability, which assesses the extent to which items within a test yield consistent results. Available 
reliability evidence for DIL assessments appears in Table 6, Column 4. Across all assessments reviewed, the majority 
of reliability estimates for the total test (coefficient alpha) range from .80 to .88, thus showing good internal consistency, 
whether for more traditional IL measures (Cameron, Wise, & Lottridge, 2007; Catts, 2005; Clark & Catts, 2007; Ivanitskaya, 
2011; Radcliff, Oakleaf, & Van Hoeck, 2015; Wise et al., 2009) or for interactive performance-based measures (Katz, 2007; 
Katz, Attali, & Rijmen, 2008; OECD, 2013c). Pilot test data for the technology-focused IC3 assessment also shows high 
internal consistency, with Kuder-Richardson-20 values ranging from .88 to .93 across administrations, and test-retest 
reliability (i.e., the extent to which examinee scores at Time 1 are correlated with retest scores at Time 2) was reported 
at .82, based on a sample of 136 retest examinees (Haber & Stoddart, n.d.). However, although the CLA+ performance 
tasks, which require an extended constructed response, achieves high reliability at the institution level (a = .84; Klein et al., 
2009), it is much less reliable at the individual level (a = .43 to .57) due to the use of a single prompt for each test taker 
(when coupled with selected-response items, the combined reliability is comparable to other assessments, with a ranging 
from .85 to .87; Zahner, 2013). 

The reliability of subscores was also examined. Subscores are generally less reliable than estimates for the total test, 
although SAILS reports that subscore reliabilities exceed .80 for all eight subdomains associated with the IL standards 
(ACRL, 2000), based on data from over 40,000 college students (Radcliff & Salem, 2006). For tests that report subscores 
associated with specific standards, subscore reliabilities (a) range from .35 to .78 for ISS (Catts, 2005), from .48 to .76 
for ILT (Cameron et al., 2007; Wise et al., 2009), and from .52 to .76 for first-year college students taking RRSA (Ivanit¬ 
skaya, 2011). Interestingly, when administered to a sample of master’s level students, the overall test (a = .78) and subscore 
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to be at Level 4 in literacy and numeracy). Educational attainment, parental 
educational attainment, and level of training have a strong positive relationship 
to scores (e.g., those adults with less than an upper secondary education are 
much less likely to score at Levels 2 and 3 compared to adults with tertiary 
education, p. 188). 
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non-Caucasian (N = 86; 157 vs. 136). However, no information given on 
English-language status of latter group (as a highly verbal test, lower scores for 
ELL students has been found before). Higher self-reported ACT scores 
corresponded with higher iSkills scores. 





Assessment Author/year Participants and sample size Reliability Validity evidence 
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lower than current community-college students, who scored lower 
than students currently at a four-year institution. Unclear 
relationship between iSkills and year in school. 
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reliabilities (a = .59 to .67) for RRSA decreased relative to the undergraduate sample (overall a = .82), whereas reliabilities 
for the self-report items increased for master’s (a = .81 to .89) as compared to undergraduate students (a = .66 to .86). A 
mixed-grade sample of students (sophomores, seniors, and graduate students) from courses in communication sciences 
and disorders showed similar reliability estimates to the master’s level sample (overall a = .79, a = .67 to .72 for subscores; 
Ratcliff, Swartz, & Ivanitskaya, 2013). The iSkills assessment does not provide subscore reliabilities, citing results from 
factor analyses suggesting a unidimensional construct (Katz, Attali, et al., 2008; see the “Construct Validity” section). Of 
course, even subscores that achieve acceptable levels of reliability might not provide information about students’ knowl¬ 
edge and skills beyond what the total test score provides (cf. Sinharay & Haberman, 2008), an issue we return to in the 
“Score Reporting Considerations” section. 

In sum, most DIL assessments reviewed demonstrate good reliability, mostly due to their reliance on multiple selected- 
response items, which tends to increase reliability relative to tests with fewer items. 

Content Validity 

Content validity evidence involves examining the correspondence or alignment of assessment items and the subject mat¬ 
ter domain being assessed (i.e., DIL). Content validity is often evaluated by asking subject matter experts or test users 
to make judgments of the alignment between test content and curriculum or standards; this type of expert review was 
conducted as a part of the validation efforts for a number of assessments. For example, assessments aligned to informa¬ 
tion literacy standards have examined university librarians’ judgments of the alignment of test items to those standards, 
with test developers using the data to revise or eliminate items; this approach was used for ILT (Cameron et al., 2007) 
and RRSA (Ivanitskaya, Laus, & Casey, 2004), both aligned to the IL standards (ACRL, 2000), and the ISS (Catts, 2005), 
which is aligned to the ANZIIL framework (Bundy, 2004). Using a slightly different approach, SAILS (Radcliff & Salem, 
2006) compared librarians’ ratings of item difficulty to statistical item difficulty estimates, reporting moderate to high 
correlations between these estimates. Technology-focused assessments also provide content validity evidence. As part of 
the design process for the IC3, information technology professionals ranked the importance and frequency with which 
each target task was required in entry level jobs (Haber & Stoddart, n.d.), and over 250 experts provided a review of the 
alignment of test items with the intended objectives, which was used to inform item selection (Certiport, 2003). Similarly, 
developers of ECDL asked subject matter experts to review items and identify those that are critical to competency in the 
domain; an evaluation group provides continued review as new items are developed and piloted (Davis & Cleere, 2003). 

With respect to the content validity of performance-based assessments, we focus on evidence associated with the iSkills 
assessment, which used an evidence-centered design approach (Katz et al., 2004; Mislevy, Almond, & Lukas, 2003) in 
which tasks were designed specifically to provide evidence of particular components of the construct (i.e., the seven task 
types described previously). Content validity was considered during the assessment development process by asking a 
panel of librarians and faculty across multiple education levels (high school, community college, and 4-year institutions) 
to review task content and scoring for 30 tasks; 26 of these were strongly endorsed by the panel, with the remaining four 
tasks being revised based on expert feedback (Katz, 2007). Further evidence of content validity for iSkills is provided by 
Ali and Katz (2010), who examined the relevance of DIL skills to business contexts by surveying a sample of 159 human 
resource consultants, finding that over half rated the elements as important or essential for new hires; further, a survey 
of 144 business school faculty revealed that despite the perceived importance of DIL skills, few of these were noted as 
frequently or always taught. Overall, the assessments reviewed took considerable effort to validate the content of the 
assessment items, in terms of either their alignment with academic standards or their correspondence with the skills that 
are required for success in the workforce. 

Construct Validity 

Construct validity concerns the extent to which a measure assesses the intended underlying theoretical construct. Few of 
the assessments we reviewed had published evidence of construct validity, in terms of examining the internal structure 
of the assessment. Construct validity evidence is provided for the ISS by way of a factor analysis conducted on responses 
from a sample of 224 first-year undergraduates (Catts, 2005); this analysis revealed six factors with Eigenvectors greater 
than 1.0, which accounted for 41% of the variance in scores; further, all items loaded on the first factor, with 24 of 37 
receiving loadings exceeding .30. The construct validity of the iSkills assessment was examined using a confirmatory factor 
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analysis (Katz, Attali, et al., 2008); this analysis revealed that a one-factor model had a better fit to the data as compared 
to seven-, four-, and two-factor theoretical models. Taken together, the limited construct validity evidence suggests that 
the DIL construct is uni dimensional; this could help to explain the low subscore reliability estimates for assessments that 
report them. 

Relation to Other Variables 

Another form of validity evidence involves inspecting the relationship between assessment performance and performance 
on other measures given at the same or similar times (i.e., concurrently). Convergent evidence is for measures expected to 
be related (either positively or negatively) to the assessment, whereas discriminant evidence is for measures expected not to 
be related to the assessment (i.e., having zero correlation). The pattern of convergent and discriminant measures provides a 
picture of the construct assessed by the assessment of interest. This pattern aids interpretation of assessment performance 
by painting a picture of the knowledge and skills that tend to be related and unrelated to the target construct. In some 
cases, there might be a theory to predict the expected relationships; in other cases, the inspection of concurrent measures 
might be more exploratory, such as when a construct that has not been extensively studied is the intended construct for 
an assessment. Across the DIL assessments we reviewed, the majority of validity evidence available concerns inspecting 
relationships between particular DIL assessments and other variables of interest, with studies reporting correlations of 
test scores with other measures of information literacy skills, such as librarian judgments (Catts, 2005), self-report items 
(Cameron et al., 2007; Ivanitskaya, O’Boyle, & Casey, 2006; Katz, Attali, et al., 2008; Katz & Smith-Macklin, 2007; OECD, 
2013c), or other DIL assessments (Beile, Dziuban, Katz, & Salem, 2010; Cameron et al., 2007; Katz, Elliot, et al., 2008; 
Klein et al., 2009; Mery, Newby, & Peng, 2011; OECD, 2013c; Project SAILS, 2015b), as well as correlations with more 
general academic measures, such as year in school (Cameron et al., 2007; Clark & Catts, 2007; ETS, 2007; Hill, Macheak, 
& Siegel, 2013; Ivanitskaya, 2011; Klein et al., 2007; Radcliff et al., 2015; Ratcliff et al., 2013), grade point average (GPA; 
Benjamin & Chun, 2003; Cameron et al., 2007; Gross & Latham, 2007; Ivanitskaya, 2011; Katz, Elliot, et al., 2008; Katz, 
Haras, & Blaszczynski, 2010; Katz & Smith-Macklin, 2007; O’Connor, Radcliff, & Gedeon, 2001), or SAT®/ ACT scores 
(Beile et al., 2010; Benjamin & Chun, 2003; ETS, 2007; Gross & Latham, 2007; Hignite, Margavio, & Margavio, 2009; Katz, 
Elliot, et al., 2008; O’Connor et al., 2001; Radcliff & Salem, 2006). 

Studies have typically found a positive relationship between DIL assessments and several general academic ability 
measures. For example, researchers have reported correlations in the .4 to .5 range between iSkills and SAT-Verbal scores 
and approximately .35 for iSkills and SAT-Math scores (Beile et al., 2010; Katz, Elliot, et al., 2008). Similarly, for SAILS, 
Beile et al. (2010) reported a correlation of .42 for SAILS and SAT-Verbal as compared to a nonsignificant correlation of 
.16 with SAT-Math, although the relationship between SAT and SAILS performance is reported to be inconsistent across 
items (O’Connor et al., 2001). The higher correlations for SAT-Verbal are not surprising given the amount of reading and 
verbal reasoning inherent in measures of information literacy, although the magnitude of these correlations suggests that 
DIL reflects a separate construct from reading and writing literacy per se. Hignite et al. (2009) similarly reported that 
higher ACT scores were related to higher iSkills scores. Examinations of the CLA+ reveal that the overall test correlates 
r = .90 with SAT at the institution level (Benjamin & Chun, 2003), while at the student level, the performance task essay 
is correlated with SAT at r = .44 and .46 for first-year and senior students, respectively (Klein et al., 2007). For the PIAAC 
Survey of Adult Skills, correlations are computed among the three scales of the survey; correlations of PS-TRE with literacy 
range from .68 to .82, while correlations between PS-TRE and numeracy (i.e., mathematics) range from .66 to .76 across 
the 19 countries that participated in the PS-TRE assessment (OECD, 2013b). In general, DIL assessments show moderate 
to high correlations with standardized cognitive assessments. 

With respect to GPA, correlations are somewhat lower; at the student level, CLA+ correlates r = .50 with GPA (Ben¬ 
jamin & Chun, 2003), as compared to a correlations of .20 between cumulative GPA and the ILT (Cameron et al.,2007) and 
.34 between GPA and RRSA performance (Ivanitskaya, 2011). Correlations between GPA and iSkills have been more vari¬ 
able, ranging from zero to moderate correlations, whether that GPA is self-reported (r= .23 in Katz & Smith-Macklin, 
2007; r= .23 for native English speakers in Katz et al., 2010) or pulled from school records (r= .08 for first-year stu¬ 
dents and r = .27 for sophomores, juniors, and seniors in Katz, Elliot, et al., 2008). Correlations between DIL assessments 
and course grades are low for iSkills (r= .24 in Beile et al., 2010; r= .21 in Katz, Elliot, et al., 2008) and are nonsignif¬ 
icant for SAILS (r= .13; Beile et al., 2010). Further, SAILS shows no relationship to high school GPA (O’Connor et al., 
2001 ). 
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Results for year in school generally show increased performance on DIL assessments with increasing school experience 
(Cameron et al., 2007; Hill et al., 2013; Ivanitskaya, 2011; Radcliff et al., 2015; Ratcliff et al., 2013); for SAILS, items related 
to accessing, selecting, and evaluating sources are most likely to show improvement from first-year to senior students 2 
(Radcliff & Salem, 2006). The RRSA shows significantly higher performance for graduate students as compared to both 
seniors and sophomores (Ratcliff et al., 2013); further, for SAILS, the type of institution matters, with students attending 
doctoral-degree granting institutions performing better than other institutions (Radcliff et al., 2015). Results from the 
PLAAC PS-TRE survey also show that examinees’ educational attainment, parental educational attainment, and level of 
training are strongly positively related to scores (OECD, 2013b). The self-report ISS is an exception to the general pattern; 
research shows that first-year medical students outperform fourth-year medical students on all subscales (Clark & Catts, 
2007), which the authors suggest reflects the fact that advanced medical students working in clinical settings use different 
resources for information gathering, such as relying on conversations with their supervising physician or other colleagues, 
rather than using digital and text sources. In general undergraduate settings, however, DIL assessments appear sensitive 
to the development of those skills throughout the college experience; results for the ISS suggest that this measure may 
not be sensitive to discipline-specific methods of accessing and evaluating information, which students may be asked to 
master as they enter into graduate or professional education within a domain. 

Other studies have reported convergent evidence that DIL assessments correlate with measures thought also to reflect 
digital literacy skills; one area that has been examined is relations with self-report measures. For example, in a study 
of iSkills, Katz and Smith-Macklin (2007) administered a self-report measure of information literacy skills, a 30-item 
questionnaire on test takers’ confidence in their ability to perform DIL activities. The activities were created based on 
the iSkills digital literacy framework. The researchers reported a correlation of r = .27 between iSkills and overall DIL 
confidence. Interestingly, simply performing these activities (i.e., being asked to rate the frequency of these activities) was 
not correlated with iSkills performance, which suggests that frequent DIL activity does not necessarily lead to stronger 
DIL skills. Katz and Smith-Macklin reported results for college students, but similar results were obtained when iSkills 
was administered to a sample of adult workers at a call center, with correlations of approximately .40 between iSkills and 
self-reported confidence (Katz, Attali, et al., 2008). Confidence ratings have also been examined for the ILT, with Cameron 
et al. (2007) reporting that ILT scores are correlated with self-reports of confidence in finding books and scholarly articles 
for research projects and confidence in source evaluation skills. Studies of the RRSA similarly report that test scores are 
positively correlated with proxy measures of library use, such as self-reported exposure to library instruction (Ivanitskaya 
et al., 2004), yet no specific correlation coefficients are reported. Among US adults, self-reports of use of information 
communication technology at home and in the workplace are correlated .33 and .22 with performance on the PIAAC 
PS-TRE (OECD, 2013c). In contrast to these self-report measures, convergent validity evidence for the ISS was provided 
by librarian ratings of students’ information literacy skills; these expert ratings were correlated r = .42 with scores (Catts, 
2005), suggesting that these experts may provide more precise estimates of DIL proficiency as compared to students’ self¬ 
perceptions; however, this is the only study that provided such evidence, and self-reports are likely easier to obtain because 
they can be presented within the same testing session as more objective items (as in the RRSA). However, Rosman, Mayer, 
and Krampen (2014) cautioned that correlations with self-report judgments might be overestimates if the self-report 
measure was administered after the information literacy assessment. 

Other validation efforts have examined correlations among DIL assessments and other measures designed to assess 
the same or similar constructs. For example, the CLA+ performance task is highly correlated (r = .73 to .83) with other 
measures of critical thinking (e.g., Measure of Academic Proficiency and Progress [MAPP]; Klein et al., 2009). The ILT 
is significantly correlated (r = .38) with the ISST, a predecessor DIL measure developed for use within James Madison 
University, on which the design of the ILT was based (Miller, 2004, as cited in Cameron et al., 2007). This correlation, based 
on a sample of 333 students who had taken both tests, improved to .45 after removing unmotivated students, identified 
on the basis of timing data. ILT is also highly correlated with SAILS (r = .67, r = .72 adjusted for reliability; Project SAILS, 
2015b), which has been shown to correlate with locally developed IL assessments (r = .53 to .67; Mery et al., 2011). Further, 
studies have found positive correlations between SAILS and the iSkills assessment (r = .56), based on a small sample of 
50 undergraduates (Beile et al., 2010). In all of these cases, the correlations suggest that existing assessments tap similar 
knowledge and skills related to DIL. 

To further examine the convergent validity of the iSkills assessment, Snow and Katz (2009) administered a set of 
iSkills “evaluate” tasks and performance on homework-like evaluation tasks. The performances of 88 undergraduates were 
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correlated at r= .19 (this increased to .40 after correcting for low reliability of the homework-like tasks). A more extensive 
study compared performance of several samples of undergraduates on iSkills and an information literacy rubric in use at 
the university (Katz, Elliot, et al., 2008). Across several samples that included first-year students, sophomores, juniors, and 
seniors, the researchers observed little connection between iSkills scores and rubric scores for students’ writing portfo¬ 
lios. The relation between iSkills scores and other academic ability measures were largely replicated as in previously cited 
studies. The researchers concluded that the type of information literacy skills tested by iSkills involves more goal-directed 
and time-limited skills compared with the semester-long portfolios and, so, taps different constructs. This conclusion 
is supported by another study that demonstrated an alignment between iSkills scores and grades in a business-writing 
course (Katz et al., 2010). Overall, then, while iSkills and other DIL assessments appear to measure a similar construct, 
these constrained instruments do not completely correspond to the range of information literacy experiences that students 
encounter in their coursework. 

Evidence Based on Test-Criterion Relationships 

An assessment may be used for placement, either with respect to placing out of an otherwise required course or being 
asked to take a remedial course in order to further develop one’s limited skills. In either case, validity evidence should be 
collected so that performance on the assessment in some way is predictive of performance in the course to be taken or 
skipped. If it can be shown that performance on the assessment taken before a course is related to students’ performance 
in the course (e.g., final course grade), it strengthens the validity argument for the use of the assessment for placement. 
A familiar example of predictive validity is the SAT exam, which is designed to predict first-year GPA in a 4-year college 
(Bridgeman, McCamley-Jenkins, & Ervin, 2000; Morgan, 1989). Although the SAT is a selection, rather than a placement 
exam, a similar validity logic holds. A typical predictive validity study for the SAT follows a cohort of students who took 
the SAT into their first year at college. Regression analyses are used to show the predictive power of the SAT score on first- 
year GPA after controlling for other factors (e.g., gender, ethnicity, parental education, family income) and in comparison 
to other possible predictors, like high school GPA. 

Across the assessments we reviewed, limited evidence of test-criterion relationships (for prediction) was available; to 
our knowledge, the iSkills assessment was the only measure for which published evidence was available. A study was con¬ 
ducted using iSkills as a predictor for performance in an analytical business writing course at a college in California. Katz 
et al. (2010) administered the iSkills assessment to two cohorts of undergraduate business majors: native English speakers 
and nonnative English speakers. Regression analyses demonstrated a significant relationship between the beginning-of- 
course iSkills scores and end-of-course grades for both cohorts. The researchers reported that iSkills accounted for about 
10% of the variability in each cohort’s final grades, after controlling for students’ general academic ability (i.e., GPA). 
This degree of prediction corresponds to students who did well on iSkills being approximately six times more likely 
to earn an A in the class compared with students who performed poorly on iSkills. This study provides validity evi¬ 
dence for the use of iSkills as a prescreener for a business writing course; for example, students who perform poorly 
might be asked to take a remedial course that covers the basics of conducting a focused search of a literature and com¬ 
municating conclusions reached through synthesis and evaluation of the identified information (i.e., skills covered in 
business writing courses). Thus, while the iSkills assessment shows some evidence of predictive validity in business writ¬ 
ing contexts, the predictive validity of other DIL assessments has been understudied (and perhaps underreported) in the 
literature. 

Summary of Validity Evidence 

Across the assessments included in the current review, several conclusions maybe drawn. First, assessments of DIL appear 
to capture the construct as defined by academic standards, workforce requirements, or other theoretical conceptions of 
the DIL construct, despite some variation in what is measured across these various instruments. Second, it appears that 
performance on DIL assessments is moderately to highly correlated with other measures of this construct, as well as more 
general measures of academic success, such as GPA or course grades. Because DIL skills are associated with academic 
success, and because these skills appear to develop over the course of students’ undergraduate and graduate training, it is 
important to obtain valid measures of these skills for use in evaluating the extent to which students possess DIL proficiency 
across various institutional contexts. The preceding review provides an overview of currently available measures that 
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institutions could choose to implement; in the following section, we turn to a discussion of important characteristics of 
and considerations for the design of a new DIL assessment, which could serve as a measure of SLOs. 

Proposed Characteristics of a Digital Information Literacy Assessment 

An operational definition of the DIL construct (like that proposed in Table 3) forms just one component of the assessment 
design considerations; beyond a specific construct definition, assessment developers must consider several other issues, 
such as the contexts in which information is to be accessed, evaluated, and used; the extent to which particular technology 
applications are included and whether use of the technology constitutes a context for problem-solving or is an assessment 
target in its own right; and considerations related to scoring. We discuss each of these issues in terms of their implications 
for design of a next-generation assessment of DIL as an SLO. Often, a primary driver of the selection among alternative 
assessments is the extent to which a given instrument aligns with the intended construct; this is particularly important 
given the evolving nature of conceptions of DIL. Given the integrative nature of our proposed DIL construct definition, 
the close alignment of the iSkills assessment design with this definition, and the extensive validity evidence available for 
the performance-based tasks used in iSkills, we primarily use the design features of that assessment as a basis for our 
discussion and subsequent recommendations. 

Defining an Evolving Construct 

Developers appear to take one of two approaches when considering how to operationalize the construct of DIL for pur¬ 
poses of developing assessments. The first constitutes identifying a particular framework or standard that aligns with the 
developers’ definition of the construct and then writing items aligned to those objectives or components of a standard. 
Instruments of this type, which are based on particular standards or objectives, would be of interest to administrators 
who want to assess a discrete set of skills. For example, a definition of information literacy as “the ability to successfully 
conduct information searches” (Rosman et al., 2014, p. 741) suggests that the use of search and retrieval tasks should be 
sufficiently representative to measure competency with information literacy. This method contrasts with those of devel¬ 
opers who operationalize the construct at a more conceptual level, thus allowing for the creation of real-world tasks that 
measure the construct more broadly and more authentically. Assessments developed from this perspective may be of 
interest to administrators who define the construct more holistically and are interested in whether students can deploy 
these skills in the context of real-world applications in which those skills are required. Ultimately, the intended learning 
objectives for a given information literacy program and the operational definition on which an assessment is based should 
be well aligned for the test to be considered appropriate. 

Possibly the most significant attempt to change the definition of information literacy has been the recently adopted IL 
framework (ACRL, 2015). In comparing the IL framework to the previous IL standards (ACRL, 2000), it is clear that an 
attempt is being made to better communicate the nature of information literacy skills. The IL standards could be inter¬ 
preted as defining information literacy as a list of component information literacy skills because each standard is followed 
by a list of performance indicators — that is, behaviors indicative of elements of information literacy competency. The 
definitions of skills and performance indicators in the IL standards might have suggested to many that these component 
pieces compose the entirety of the information literacy construct. Perhaps unintentionally, such an atomistic approach to 
standards suggests a particular way of teaching information literacy, through the incremental and piecemeal accretion of 
the individual skills. That is, to teach information literacy, students must learn each of the constituent parts as outlined 
in the IL standards. For example, Katz and Elliot (2016) described this viewpoint in the way that educators wanted to use 
the iSkills assessment. Some educators wanted scores on subskills within the assessment to know which subskills their 
students were weak on, so that they could then increase instruction on those subskills. This approach represents both an 
inappropriate use of the test (subscores are not sufficiently reliable and highly intercorrelated), as well as treating DIL as 
a set of distinct subskills, and fails to properly take into account the integrative, purpose-driven nature of the construct. 
The whole of information literacy is greater than the sum of the parts as listed in the IL standards. 

In contrast, through the updated IL framework, the ACRL appears to have adopted a less atomistic and more holistic 
view of information literacy (and, by extension, DIL). The IL framework is organized around six “big ideas,” which are 
deliberately described at a high level, perhaps in the hope that instructors will be forced to think deeply about the big ideas 
and how those ideas might be taught, and internalized, by students. These big ideas reflect a more integrative organization 
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of the IL skills encompassed by the original standards, and accordingly, some current DIL assessment developers have 
already proposed alignments between their test and the six big ideas from the IL framework (e.g., ILAAP, 2015). Other 
assessments remain relevant because they do not overrely on the specific definitions and performance descriptors apparent 
in the earlier IL standards. For example, the alignment between the IL standards and the iSkills test was deliberately done 
not at the level of the individual tasks, but between the standards and descriptions of performance indicators (i.e., the 
observable behaviors elicited by tasks as evidence of knowledge and skill; Katz et al., 2004). These performance indicators 
motivated the construction of iSkills tasks as well as the design of the scoring algorithms for each task. Thus, it is only in 
the broadest sense (the task types) that the tasks are aligned to the original IL standards. 

As performance tasks, the iSkills tasks were designed to embody rich, interactive performances that capture a range 
of lower level skills, but in an integrated way, that reflects how information literacy skills and knowledge are used in 
real-world tasks. This approach — from the design of the assessment right through to the scenario-based nature of the 
tasks — differs from those of most existing DIL assessments. The iSkills tasks were designed to elicit certain behaviors that 
are indicative of pieces of the construct, but the assessment is at the level of the entire construct. The iSkills assessment 
differs from a multiple-choice test, even one based on IRT, because the test is not designed to have one-to-one correspon¬ 
dences between the individual items and pieces of the IL standards, but rather to integrate skills that cover several elements 
of the standards as is natural in more real-world performances. It is true that the iSkills tasks fall short of true real-world 
performance. For example. Snow and Katz (2009) demonstrated how test takers spent more time trying to figure out 
what an iSkills task was asking them to do compared with a more direct focus on the solving the task that was seen in 
the verbalizations of test takers when they solved parallel, but more naturalistic, homework-like tasks. Nevertheless, the 
iSkills tasks, by creating a scenario-based context and requiring simulated performances, seems to reflect a more holistic 
view of the information literacy construct compared with tests that require relatively simpler actions (e.g., selecting from 
options, moving objects on the screen, or demonstrating a particular path to do something in simulated software), and 
whose items are designed to have 1:1 mappings with elements of the IL standards. 


Real-World Scenarios 

DIL are skills applied in the real world. In the workplace, school, and personal lives, we engage with information in the 
context of digital tools and media. This engagement is necessarily interactive: we seek information, we learn about a topic 
and thereby change the way we look for further information, and we make decisions about the quality of different infor¬ 
mation sources that affects our interpretation of the information we get from those sources. To align with the interactive 
nature of the domain, an authentic assessment should be similarly interactive, or performance-based, and embedded in a 
real-world context; this approach is exemplified by the interactive tasks included in the PS-TRE and iSkills assessments. 

The real-world scenarios should reflect the range and complexity of contexts in which DIL activities occur, whether 
in the workplace, through academic work, or in one’s personal life. Workplace scenarios might focus on creating analytic 
syntheses of business information, perhaps to support decision-making (e.g., a memo describing characteristics of a com¬ 
petitor’s products). School-based scenarios might include typical classroom assignments encountered in college, such as 
evaluating alternative sources of information to use in a writing assignment or creating a presentation to summarize a 
research project to one’s classmates. Personal life scenarios might include such activities as conducting internet searches 
to help a family member solve a problem or reorganizing files to make them easier to find later. For example, a recent 
experience-sampling survey conducted by the Pew Research Center (A. Smith, 2015) reveals that a majority of younger 
adults ages 18-29 report using their smartphones for various everyday purposes, including accessing information about 
health conditions, real estate, and employment opportunities, in addition to educational content. These real-world sce¬ 
narios that are relevant to the ways in which college-aged populations access and make use of digital information can help 
motivate examinees’ interactions with such information, providing a setting in which access, evaluation, and use is driven 
by specific, meaningful goals. 

A challenge to providing real-world scenarios is the rapid pace of changing technology. Scenarios that seemed current 
just afewyears ago (e.g., instant messaging or e-mail) often are replaced by more current technologies (e.g., text messaging, 
video messaging). Although the underlying cognitive skills associated with DIL might not change as rapidly as we discuss 
here, assessments should be subject to periodic reviews to ensure that the technologies and situations depicted remain 
current and relevant to higher education students. 
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Focus on Measurement of Cognitive Rather Than Technology Skills 

The review of frameworks presented earlier suggests a wide range of views on DIL, from a primarily technical set of skills 
(i.e., technical computer literacy) to largely cognitive skills that need not be associated with use of digital technology (i.e., 
traditional information literacy). Our proposed operational definition (Table 3) bridges these approaches, focusing on 
measuring cognitive skills in the context of digital technology applications rather than measuring use of technology for 
its own sake or cognitive skills devoid of a technology-infused environment. This approach has the advantage of being 
relatively impervious to rapid changes in technology. Although recent years have seen large changes in technology and the 
technological contexts of engaging with information, the cognitive skills involved in appropriate use of information have 
remained largely stable since the first definition of information literacy appeared in the 1970s (Zurkowski, 1974). What has 
changed considerably are the ways in which we define or enumerate those skills. As previously described, the early ACRL 
(2000) definition implied a somewhat elemental approach to information literacy as a set of specific skills to be acquired. 
The more recent IL framework (ACRL, 2015) takes an approach more akin to Bruce’s (1997) view of information literacy 
as a set of increasingly sophisticated views on information, which are mutually supportive. That is, the more recent IL 
framework recognizes and emphasizes the integrative (vs. elemental) view of information literacy. 

An assessment should similarly represent such an integrative view of digital information, rather than treating each 
item as measuring a single, isolated assessment target. Tasks might involve extended performances that engage several 
aspects of DIL skills over multiple actions or activities. Of course, for the purposes of designing an instrument with good 
construct validity, we must have some documentation of construct coverage so tasks could be assigned to particular aspects 
of the DIL construct. However, rather than focusing very tightly on low-level skills, the assessment tasks should take a 
broader view of the domain and reflect the contexts in which application of DIL skills is required. Assessment items 
that target a shallower understanding of the domain are more typical of multiple-choice or simple constructed response 
testing formats, as opposed to richer, more extended interactions with simulated software. We advocate the use of these 
simulation tasks to obtain better evidence of students’ ability to deploy DIL skills in the technological contexts of their use. 

Use Minimal and Highly Familiar Technology Elements 

Because technology per se is not a focus of a DIL assessment, knowledge of particular applications or particular software 
functions should not be necessary to do well on the assessment. An assessment should assume only that test takers are 
familiar with broad classes of software applications (word processors, browsers, presentation tools), software functions 
(copy-and-paste, menus, file management, clicking), and hardware (mouse and keyboard) rather than a specific tool or 
application. This baseline level of expected technology know-how should be made known to test takers to allow them to 
prepare for the assessment. 

Such a minimalist focus on technology has some benefits. Many of these basic software functions have been part of 
consumer experience with technology, almost unchanged, for several decades (e.g., many tools share common graphical 
interfaces or similar design language, such as menu commands and toolbar icons). In other words, these functions rep¬ 
resent aspects of technology that have remained remarkably stable despite the overall fast-paced, changeable nature of 
technology. Thus, the minimalist functionality of software will help the stability of the assessment; it might not be neces¬ 
sary to change the assessment to accommodate new technology nearly as often as would be needed for an assessment that 
targets specific technological knowledge (e.g., knowledge of particular software applications). 

In addition to this logistical benefit, minimalist technology may support test fairness. Even if technology is ubiquitous, 
people differ in their familiarity with particular software applications and interaction techniques. For example, Pew sur¬ 
veys reveal that 85% of US adults are online and 67% of adults use smartphones (Duggan, 2015). Despite this prevalence 
of access, some adults are heavily dependent on particular methods of going online—such as via their smartphones — and 
have limited means for accessing online content from other devices (i.e., laptop or desktop computer with a broadband 
internet connection). Such “smartphone dependent” individuals represent approximately 19% of the adult population, 
including 15% of adults age 18-29 (A. Smith, 2015). These results suggest that many college-aged students (i.e., potential 
DIL examinees) may have little experience with using a desktop computer as a primary means of going online. Further, 
smartphone dependence disproportionately affects low-income (13% for household incomes under $30,000 vs. 1% for 
household incomes over $70,000) and non-White adults (12% for African Americans and 13% for Latinos vs. 4% for 
White respondents). 
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Caution should be taken so that the technology aspects of a DIL assessment do not favor or hinder performance simply 
because a person is more familiar with a particular browser or other software environment (Katz & Gorin, 2016). Simi¬ 
larly, the changing nature of access to digital information also suggests that such assessment should take care not to limit 
the design of tasks to those that can be accomplished using a desktop computer, but should also consider the extent to 
which smartphone or tablet-based interfaces requiring touch-screen technology might also be incorporated, which would 
increase the likelihood that test takers from various backgrounds would consider the context of the tasks to be sufficiently 
familiar that they can engage with the assessment. Minimalist technology in a DIL assessment helps achieve this fairness 
goal by attempting to put all test takers on a similar level of using software that has minimal, but recognizable functions, 
and helps ensure that examinees with knowledge of particular software packages or types of hardware do not have an 
unfair advantage. 

Automated Scoring 

Some open-ended DIL assessments are scored by human raters using a well-defined rubric. For example, the UK Key 
Stage 3 ICT exam presents students with contextualized real-world tasks (United Kingdom Department for Education, 
2013). The students use simulated, but highly functional software, to produce digital artifacts such as word processed 
documents or spreadsheets. These rich digital artifacts are scored by human raters. This approach to large-scale, human- 
scored performance assessment has been implemented by some national testing programs, such as the UK Qualification 
Authority (Ripley, 2009) and the New Zealand National Educational Monitoring Project (J. Smith, Crooks, & Allan, 2010). 
Some universities have also taken this approach for information literacy assessment (e.g., New Jersey Institute of Tech¬ 
nology [NJIT]; Scharf, Elliot, Huey, Briller, & Joshi, 2007). However, despite the possible benefits in terms of measuring 
complex constructs, the approach is costly and time-consuming, requiring human raters to be trained, calibrated, and 
dedicated to spending considerable time in scoring. Automated scoring might be more feasible solution for most colleges 
and universities. 

Automated scoring no longer implies multiple-choice or simple constructed response items; several of the DIL assess¬ 
ments reviewed incorporate some type of performance assessment that is nevertheless automatically scored; assessments 
such as PIAAC, iSkills, ECDL, and IC3 are all automatically scored, yet these assessments include performance tasks, such 
as having test takers interact with simulated software. Importantly, automated scoring is applied to both the response data 
(e.g., the selection of a particular alternative in a drop-down menu) and the process data generated in the course of mak¬ 
ing those responses (i.e., the sequence of actions that precedes submission of a final response; see Katz & Gorin, 2016). 
For example, in an IC3 task that requires examinees to print a document, alternative steps toward printing a document 
are all scored as correct, such as typing the correct sequence of keyboard commands or selecting the appropriate com¬ 
mands from a menu (e.g., File —► Print, etc.; Haber & Stoddart, n.d.). Similarly, the sequence of actions affects scoring in 
the iSkills assessment. For example, in a Web search task, examinees receive more credit for using sophisticated search 
strategies (i.e., joining two keywords with AND) on their first search attempt and less credit if they discover and deploy 
this strategy only after having previously submitted several rounds of less sophisticated search queries (Katz et al., 2004). 
While it is true that these performances might not be as open ended as real-world tasks, they may be sufficiently rich so 
that they provide measurement beyond that provided by traditional testing formats. 

Score Reporting Considerations 

“Score reports are where the rubber hits the road in the validity argument for an assessment” (Zapata-Rivera & Katz, 2014, 
p. 442). Assessment results drive academic decisions about institutions or individuals, and so the nature of the informa¬ 
tion reported from an assessment strongly determines that assessment’s utility for a particular institutional purpose. When 
selecting an assessment, institutions should consider not only the characteristics of the assessment itself—the target con¬ 
struct, the testing format, the test length—but also the nature of the information provided about student performance. 
The nature and type of score reports available vary across DIL assessments, with some tests only providing information to 
institutions about aggregate groups of students, whereas others provide individual-level score reports with personalized 
feedback and subscores. Institutional-level information might inform the overall performance of groups of students (e.g., 
particular majors or grade levels) and might be used to determine whether an institution has achieved its accreditation 
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goals for DIL SLOs. Individual-level information might help students determine whether further coursework or other 
instruction is needed. 

Ideally, a DIL assessment would provide useful information at both the institutional level and individual levels. In 
either case, score providers should ensure the quality of the information provided in score reports. For example, when 
providing subscores, one should ensure that these subscores are both reliable and distinct from each other (Sinharay & 
Haberman, 2008). If the subscores are highly correlated, even if they are reliable, it may make less sense to report them as 
the subscores may not provide information beyond what is provided in the total scores. Further, we recommend that at the 
individual level, institutions seek reports that include performance feedback: descriptive, qualitative information about 
students’ performance on DIL tasks, rather than atomistic, quantitative information, to help score users better understand 
students’ strengths and weaknesses for improvement purposes. 


Conclusions 

DIL is an increasingly important set of skills in today’s environment of ever-changing access to and interactions with tech¬ 
nology, particularly in light of the centrality of technology for contributing to the emerging global information economy. 
Because DIL skills are important for professional and academic success, higher education institutions should consider 
the extent to which their students possess these skills yet also recognize that the construct of DIL reflects the coordina¬ 
tion of cognitive skills, thereby extending beyond the ability to use specific technologies for their own sake. The evidence 
presented in this report reveals considerable continuity among definitions of DIL, as representing a set of integrated infor¬ 
mation processing skills that are distinguishable from measures of general academic ability (e.g., general literacy skills). 
Thus, DIL is a distinct construct that can be assessed in a variety of ways. The current review presents a number of available 
assessment options that higher education institutions may use to obtain evidence of this construct, including self-report, 
selected-response and performance-based measures, although those assessments demonstrate varying degrees of reliabil¬ 
ity and validity evidence, as shown here. We suggest that a performance-based assessment of DIL that presents information 
problems in a wide range of personal, workplace, and academic contexts and aligns to widely accepted aspects of the con¬ 
struct appears to be a particularly useful option for meeting the needs of higher education institutions. We hope that this 
paper serves as a resource to this community in selecting among existing DIL assessments or in designing new approaches 
to DIL measurement. 

Notes 

1 The European Computer Driving Licence (ECDL) was initially launched in Europe in 1996. Three years later the assessment was 
offered internationally and is known as the International Computer Driving Licence (ICDL) outside of Europe (ECDL 
Foundation, 2015a). We use ECDL throughout the paper. 

2 Note that accessing, selecting, and evaluating sources are not strictly related to contexts of digital information use (i.e., students 
must deploy these skills when selecting among available print resources in libraries). As SAILS includes questions that measure 
information skills in both print and digital contexts, it is not clear from these data whether the reported improvements are due 
specifically to the development of DIL skills, or more general competency with library research methods. 
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